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Abstract The ROC curve and the corresponding AUC are popular tools for the 
evaluation of diagnostic tests. They have been recently extended to assess prognostic 
markers and predictive models. However, due to the many particularities of time- 
to-event outcomes, various definitions and estimators have been proposed in the 
literature. This review article aims at presenting the ones that accommodate to right- 
censoring, which is common when evaluating such prognostic markers. 



1 Introduction 

In the medical literature, a variety of general criteria have been used to assess di- 
agnostic tests l24l [141 . Among them, the ROC curve and the area under it - the 
AUC - are popular tools, originally aimed at evaluating the discriminant power 
of continuous diagnostic tests. In this simple situation, the outcome status D is a 
binary variable (typically, D = 1 for cases and D = for controls) and the ROC 
curve for a continuous diagnostic test X plots the true positive rate, or sensitivity, 
TPR(c) = P(X > c\D =1) against the false positive rate, or one minus the speci- 
ficity, FPR(c) = P(X > c\D = 0), when making threshold c vary. The AUC, which 
is the area under this curve, is a commonly used summary measure of the informa- 
tion contained in the sequences (TPR) c gR and (FPR) c£ r. As such, it inherits some 
of the properties of true and false positive rates. In particular, it is not affected by 
the disease prevalence (unlike positive and negative predictive values) and it can be 
evaluated from random samples of cases and controls. It also has a nice interpre- 
tation since it corresponds to the probability that the marker value of a randomly 
selected case exceeds that of a randomly selected control. 
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The extension of the AUC (and other evaluation criteria) to the setting of prognostic 
markers has raised several issues. In particular, when evaluating such markers, the 
outcome status typically changes over time: in a cohort study for instance, patients 
are diseased-free when entering the study and may develop the disease during the 
study. This leads to three major differences with the evaluation of diagnostic tests. 
First, this time-dependent outcome status (which may be defined in several ways, as 
will be seen in Section[5]i naturally implies that sensitivity, specificity, ROC curves, 
their AUC values and, more generally, any extension of the criteria used in the di- 
agnostic setting, are functions of time as well. Second, the time-to-event, i.e. the 
time between the entry in the study and the disease onset, is usually censored and 
not fully observed, requiring dedicated inference. Third, the time lag between the 
entry in the study and the disease onset also leads to two further refinements: (/) the 
marker can be repeatedly measured over time and (ii) competing events (in addition 
to censoring) may be observed between the entry and the putative disease onset. 
These particularities has led to the development of numerous methods aimed at es- 
timating the time-dependent AUC for prognostic markers. In this paper, we review 
those that accommodate to right-censoring. Some notations are introduced in the 
following Section H] Then, in Section [3] we will present several definitions and es- 
timators of the time-dependent AUC in the "standard" setting of a baseline marker 
and univariate survival data. Section |4] will cover the case of longitudinal markers 
which corresponds to the marker being repeatedly measured over time, while we 
will discuss the competing events setting in Section [5] Finally, concluding remarks 
will be given in Section|6] 



2 Notations 

Let 7] and C, denote survival and censoring times for subject i, ; = 1 , . . . , «. We fur- 
ther let Zj = min(7/,C,) and <5, = 1(7; < Q) denote the observed time and the status 
indicator respectively. We will denote by D, (? ) the time-dependent outcome status 
for subject i at time t, t > 0. Several definitions for A(?) will be given hereafter, 
but we will always have Dj(t) = 1 if subject i is considered as a case at time t and 
Dj(t ) = if subject i is considered as a control at time t. 

We will denote by X the marker under study, which can be a single biological marker 
or several biological markers combined into a predictive model (in this case, it is 
assumed throughout this article that the predictive model has been constructed on an 
independent data set; otherwise sub-sampling techniques are needed ||2TI ). Without 
loss of generality, we will suppose that larger values of X are associated with greater 
risks (otherwise, X can be recoded to achieve this). We will denote by g and G _1 
the probability density function and the quantile function of marker X. In Section 
[3] we assume that marker X is measured once at t = 0, and we will denote by Xi 
the marker value for subject i. In Section|4] which treats the longitudinal setting, the 
marker is measured repeatedly over time, and we will denote by Xi(s) the marker 
value at time s for subject i. 



Time-dependent AUC with right-censored data: a survey study 

3 Time-dependent AUCs in the standard setting 



3 



Definitions of time-dependent ROC curves, ROC(f), follow from definitions of 
usual ROC curves and thus rely on first defining time-dependent true and false posi- 
tive rates. For any threshold c, these two functions of time are defined as TPR(c, t ) = 
P(X > c\D{t) = 1) and FPR(c,f) = P(X > c\D(t) = 0). ROC(f) then simply plots 
TPR(c,f) against FPR(c,f) making threshold c vary. The time-dependent AUC at 
time t is then defined as the area under this curve, 



As a matter of fact, these definitions deeply rely on that of the outcome status at time 
t , D(t). Heagerty and Zheng lfl8l described several definitions of cases and controls 
in this survival outcome setting. According to Heagerty and Zheng's terminology 
and still denoting by 7) survival time for subject i, cases are said to be incident if 
T, ■ = t is used to define cases at time ?, and cumulative if 7} < t is used instead. 
Similarly, depending on whether 7] > T for a large time T > t or T[ > t is used for 
defining controls at time t , they are said to be static or dynamic controls. Depending 
on the definition retained for cases and controls at time t, four definitions of the 
time-dependent AUC value may be put forward. In the following paragraphs, we 
will present formulas and estimators for the most commonly used ones and will 
discuss their respective interests. 



The setting of cumulative cases and dynamic controls may be regarded as the most 
natural choice for planning enrollment criteria in clinical trials or when specific 
evaluation times are of particular interest. It simply corresponds to defining cases at 
time t as subjects who experienced the event prior to time t, and controls at time t as 
patients who were still event-free at time t. In other words, it corresponds to setting 
£),(?) = 11(7/ < f). Cumulative true positive rates and dynamic false positive rates 
are then respectively defined as 



TPR L (c,f) =P(X >c\T <t) and FPR (c,f) = P(X > c\T > t). (2) 



The cumulative /dynamic AUC at time t is then obtained by using these definitions 
of true and false positive rates in (HJ. Usually, however, 1(7 < t ) is not observed for 
all subjects due to the presence of censoring before time t and simple contingency 
tables can therefore not be used to return estimates of TPR C (c,f), FPR D (c,t) and 
AUC C ' D (f ). To handle censoring, Bayes' theorem can be used to rewrite AUC C ' D (f) 
as a function of the conditional survival function P(7 > t\X = x) (see paragraph 
|3.1.1| below). Other approaches rely on so-called Inverse Probability of Censoring 




(1) 



3.1 The cumulative dynamic AUC: AUC ' (t) 
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Weighted (IPCW) estimates (see paragraph 13.1.21 below). Before describing these 
two approaches in more details below, we shall add that Chambless and Diao J6) 
developed an alternative method - which will not be described here - based on an 
idea similar to the one used to derive the Kaplan-Meier estimator of the cumulative 
distribution function in the presence of censoring. Among all these methods, only 
those relying on primary estimates of P(T > t\X = x) (and a recent extension of 
IPCW estimates proposed in |2|]) may account for the dependence between censor- 
ing and the marker (since they basically only assume that T and C are independent 
given X and not that T and C are independent). We refer the reader to JJJUSIE) f° r 
empirical comparisons and illustrations of these various methods. 



3.1.1 Methods based on primary estimates of P(T > t \X = x) 

Bayes' theorem yields the following expressions for TPR c (c,f) and FPR D (c,f) 

V/ , r°°P(r < t\X = x)g(x)dx n , s [°°P(T > t\X =x)g(x)dx 
TPR C c,f =- — =-r x > FPR D c,f) = - — r — 

y 1 ' p(t <t) ' v ' ; p(r >f) 

From ([T), it readily follows that 

' J-ooJc P(T<t)P(T>t) «wsw 

Since P(T > t) = /_! 00 P(7 , > t\X = x)g(x)dx, any estimator S„(t\x) of the condi- 
tional survival function P(T > t\X = x) yields an estimator of AUC C ' D (f ): 

= gti r i=l s„ (t \xj)[i - s n ( t \mm > xj) 

LUr j=l s n (t\Xj)[i-s n m)} 

In JS), the authors suggested to use a Cox model to derive estimates S„(t \x), while 
one of the methods described in Heagerty et al. ifTTl reduces to using the conditional 
Kaplan-Meier estimator as in [Q] . Some theoretical results for these two methods can 
be found in 11331 and |3]|2ll2Ql respectively. 

We shall add that Viallon and Latouche J53 related AUC C ' D (f) to the quantity 
P(T <t\X = G~ l (q)) - a time-dependent version of the predictiveness curve: 

AUL {t) ~ P(r> f )p(r<r) 

This confirms that most standard statistical summaries of predictability and discrim- 
ination can be derived from the predictiveness curve, as pointed out in lfl4lll5l . 
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In |fl9l and [36], the authors independently suggested to use IPCW-type estimates: 

TPR c (c ,o = LLll(x,>c - Zi -^, m\c lt )=^£*> e > z '> t \ 

where Sc(') is the Kaplan-Meier estimator of the survival function of the censoring 
time C. The expression of the false positive rate estimator is more compact because 
weights all equal 1 / (nSc{t)) under the assumption of independence between C and 

- — - D 

X, and then vanish. FPR (c,f) corresponds to 1 minus the empirical distribution 
function of X among individuals for whom Z, > t . In the absence of censoring before 

time f, TPR (c,t) also reduces to the usual empirical version of TPR c (c,f), i.e., 1 
minus the empirical distribution function of X among individuals for whom 7} < t . 
It can be shown (see £l9]|30]) that an estimator of AUC C D (r) is then given by 

AUC ' (/) = *cWJ4cW 



n 2 S(t)[l-S(t)} 
where S(t) is the Kaplan-Meier estimator of P(T > f). 

Theoretical guarantees for these estimators can be found in 11911 and 1361 , These es- 
timators are in a sense more flexible than those presented in paragraph ^ . 1 . II above: 
they are model-free and they do not rely on any bandwidth selection (unlike the 
estimator of Heagerty et al. ifTTIl for instance, which is based on a local version of 
the Kaplan-Meier estimator). However, when censoring may depend on marker X, 
quantities like the conditional survival function of C given the marker X, Sc(-\X), 
have to be estimated |2], which implies either to work under some (semi-)parametric 
model or the selection of some parameter if nonparametric estimation is prefered. 



3.2 The incident static AUC: AUC Kh (t) 

Using the dynamic definition of controls, the control group varies with time, and so 
does the x-axis of the corresponding ROC curves: in situations where trends over 
time are of particular interest, this renders their interpretation more difficult (since 
such trends may be partly due to changing control groups). Moreover, the group of 
static controls is interesting in that it tries to mimic the group of individuals who 
never develop the disease, which can be seen as an ideal control group in some 
situations. In particular, patients with preclinical diseases are eliminated from the 
control group as far as possible, if T is large enough. 

Regarding cases, the incident definition has several advantages over the dynamic 
definition 126]. The cumulative TPR does not distinguish between events that occur 
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early versus late, and it shows redundant information over time (since early events 
are also included in the cumulative TPR for late evaluation times). Moreover, as 
pointed out by |4j, the cumulative TPR can be computed from the incident TPR 
when the distribution of the event time is known. 

Putting all this together, several authors have proposed estimators of the time- 
dependent ROC curve relying on the incident definition of cases and static definition 
of controls. Standard numerical integration techniques are then used to compute an 
estimate for AUC I,s (f ) from the estimators of the ROC curve. 

Incident true positive rates and static false positive rates are defined as 

TPR I (c,/)=P(X>c|T'=/) and FPRf (c) = P(X > c\T > t). (4) 

Applying Bayes' theorem, they can further be rewritten (see, e.g., J33)) 

TPR , r)= ntW and FpRHc , S?*(T>T\X=x) g (x)dx 
l ' J JZof(t\K)g(x)dx z[> .P 00 P(T>T\X=x)g(x)dx' 

where f(t\x) = dP(T < t\X = x)/dt is the conditional density function of T given 
X = x. Under a standard Cox model of the form X(t;X) = Ao(f)exp(j3X) - here 
X{t;X) stands for the conditional hazard rate of T given X while Xq is the unspeci- 
fied baseline hazard rate - Song and Zhou ll33l deduced that 

TPR i( c t \ = / c °° ex P(/fo)exp{-Ao(Oexp(/3x)}g(x)rfx 
/Too ex P(j3jc)exp{— Aa(t)exp(l5x)}g(x)dx 
exp{-Ao(T)exp(l5x)}g(x)dx 



FPRf(c) = 



/-«, ex P{ -A) (t) exp(j3x) }g(x)dx ' 



where Ao(f ) = f'_„X{)(u)du is the cumulative baseline hazard function. Estimation 
of TPR 1 (c,t ) andFPRf(c) can then be achieved by plug-in methods. We shall add 
that Song and Zhou actually considered a slightly more general set-up where addi- 
tional covariates can be accounted for. 

In |[l"8l . Heagerty and Zheng adopted a slightly different approach. To estimate 
TPR I (c,f), they used a (possibly time-varying-coefficients) Cox model of the form 
X(t;X) = Xo(t)exp(j5(t)X) in combination with the fact that the distribution of X- 
exp(PX) for subjects in the risk set at time t is equal to the conditional distribution 
of X given T = t (see, e.g., (38)). Setting R(t) = {i : Z; > t }, this leads to 

— I, LieR( t )m>c)ew{P(t)Xi} 

As for the estimation of FPR^(c), they proposed a model-free approach using the 
empirical distribution function for marker values among the control set S T := {i : 
Z, > t}. Namely, denoting by n T the cardinality of S T , they proposed 
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n * its, 

which is FPR (c,t) of Section 13. 1.21 except T is used instead of t . 
Cai et al. J4) proposed another approach in the context of longitudinal markers; 
it will be described in more details in Section H In addition, two non parametric 
approaches were recently proposed (see |[32l and |29)). 

Note also that estimators for the time-dependant incident/dynamic AUC, AUC 1,11 (t ), 

— s / 

can be obtained by simply replacing T by t in the definitions of FPR T (c) above 
fT8l . A global accuracy measure has further been derived from the definition of 
AUC ID, (f), which is particularly appealing when no a priori time t is identified 
and/or when trends over time are not of interest lfl"8l . 



4 Time-dependent ROC curve and AUCs with longitudinal 
marker 

In this section, we review extensions of the above estimators for longitudinally col- 
lected subject measurements. For instance some authors would assess the discrim- 
ination performance of CD4 counts repeatedly measured every week on time from 
seroconvertion to progression to AIDS [42]. Therefore, time-dependent sensitivities 
and specificities have been extended to deal with the fact that (;) the time at which 
marker X is measured can vary and (ii) marker can be repeatedly measured on the 
same subject. Let s denote the timing of marker measurement and X(s) the marker 
value at time s. For t > s, Zheng and Heagerty (45) extended cumulative! dynamic 
definitions 

TPR c (c,s,f) = P(X(j) > c\T e [s,t]), FPR D (c,s,f) = T(X(s) > c\T > f). 

For a fixed time T > s, Zheng and Heagerty iffOl extended incident/static definitions: 

TPR^cM) =V(X(s) >c\T =t) and FPR s (c,s, t) = P(X(s) > c\T > z). 

Although others approaches have been proposed to estimate these quantities, we 
only review estimators that deal with censored data here. We should also mention 
that in this longitudinal context, estimators of the AUC are obtained by numerically 
integrating estimators of the ROC curve. 



4.1 Cumulative dynamic estimators with longitudinal marker 

Rizopoulos ll27l recently proposed a joint modeling approach. The marker trajectory 
is modeled by usual linear mixed model for longitudinal data, and a parametric pro- 
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portional hazard is used to model the time-to-event given the marker trajectory. The 
two submodels are linked with shared random effects to capture the intra-subject 
correlation. Standard maximum likelihood estimation is used to fit the joint model. 
Then, TPR C and FPR D are computed from the estimated parameters and Monte 
Carlo simulations are used to make inference. As this approach is fully parametric, 
its main advantage is its efficiency. This approach also allows censoring to depend 
on the marker (35] . The counterpart is that the parametric model must be carefully 
chosen, and checking model fit is not straightforward. 

A more flexible methodology was proposed in ll42l . with fewer parametric assump- 
tions. Setting T* = T — s, the "residual failure time", and t* =t — s, they rewrote 

v ' ' ' Y-S(t*\s) V ' S(t*\s) ' 

with Fx\ s {c) = P(X(s) < c\s,T* > 0) the conditional distribution of marker given 
measurement time, 5(f*|i) = P(T* > t*\s,T* > 0) the survival probability for in- 
dividuals who survived beyond s and S(c,t*\s) = T(X(s) > c,T* > t*\s,T* > 0). 
They proposed to estimate F x \ s (c) with the semiparametric estimator proposed by 
Heagerty and Pepe |fl6l . Therefore, only the location and scale of the conditional 
distribution of marker given measurement time are parametrized. To estimate the 
survival terms S(c,t*\s) and 5(f*|i), they proposed the use of a "partly conditional" 
hazard function to model the residual failure time T* = T — s. For subject i at mea- 
surement time Sjfc, this function is modeled by 

^ k (t*\Xi( Sik ),0 < s ik < Ti) = A (r*)exp [p{t*)Xi{s ik ) + a T f(s ik )] 

where f(s) are vectors of spline basis functions evaluated at measurement time s, 
and Ao(f*) is left unspecified. Estimators of /}(•) and a have been previously pro- 
posed BTI . As this approach is semiparametric, its main advantage is its flexibility. 
However, by contrast to the approach of Rizopoulos 12711 . this one is less efficient 
and does not allow marker-dependent censoring. 



4.2 Incident static estimators with longitudinal marker 

Several authors consider the incident/static definition of AUC BTI [T2l l4l [34l . How- 
ever, censored data are only accounted for by Cai et al. fij who proposed to model 

TPR 1 (c,s,t)=g D (T] a {t,s)+h(c)), t < T 
FPR s (c,s,x)=g T (t; a (s)+d(c)) 

where go and g T are specified inverse link functions and h(-) and d(-) are unspec- 
ified baseline functions of threshold c. The dependence in time is parametrically 
modeled by rj a (t,s) = a T rj(t,s) and % a (s) ~a T £, (s) where rj(t ,s) and £, (s) are vec- 
tors of polynomial or spline basis functions. These models are semiparametric with 
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respect to the marker distribution in cases and nonparametric in regards to controls. 
As pointed out in ||26l , this model is very flexible as it does not specify any distribu- 
tional form for the distribution of the marker given the event-time, but only model 
the effect of time-to-event on the marker distribution with a parametric form. Model 
estimation is performed by solving some estimating equations and large sample the- 
ory was established allowing a resampling method to construct confidence bands 
and make inference H). Interestingly, the authors of J4| also showed that covari- 
ates can easily be included in TPR 1 and FPR S , enabling to directly quantify how 
performances of the marker vary with these covariates. 



5 Time-dependent AUC and Competing risks 

We now consider the setting where a subject might experience multiple type of 
failures: in this section, we review extensions of time-dependent AUCs to competing 
risks. For example, we may want to assess the discrimination of a given score on 
death from prostate cancer with death from other causes acting as a competing event. 
For the sake of simplicity, we will assume there are only two competing events, 
and we let Sj = j denote that subject i experienced the competing event of type j 
= 1, 2, with j = 1 for the event of interest). The observed data consists of a failure 
time and a failure type (Z,-, 5,-) with <S, = denoting a censored observation. 
In their review paper l26l sketched most potential extensions and introduced event- 
specific sensitivity and specificity. They also highlighted that the crucial point was 
to determine whether patients experimenting a competing event should be treated as 
a control when evaluating the discrimination of the marker under study with respect 
to the event of interest. More precisely, two settings can be considered. 

First, if marker X is potentially discriminatory for both the event of interest and the 
competing event, then both event specific AUCs should be considered simultane- 
ously l28l FPU . For illustration, in the cumulative/dynamic setting, cases at time 
t can be stratified according to the event type, Casei = {?' : 7} < t, Si = 1} and 
Case2 = {i :Tf<t, 5, = 2}, and controls at time t are the event-free group at time f, 
Control={; : Tj > t }. Following these lines Saha and Heargerty |28] proposed event 
specific versions of (f2]i 

TPRj (c, t) =V(X > c\T <t, 8 = j), FPR n {c,t)=P(X>c\T>t,Se{l,2}). (5) 

Estimation follows from 1 17!], using the conditional cumulative incidence associated 
to competing event j, P(T <t,8 = j\X), instead of the conditional survival function 
of P(r <t\X). In the context of renal transplantation, Foucher et al. |[T3l considered 
a slight modification of definitions (O, where controls can also be "event specific". 
In addition, an extension of the incident/dynamic AUC to the competing events 
setting was proposed by Saha and Heargerty l28l . 
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The other option is to consider both event-free patients and patients with the compet- 
ing event [ 39 , 2 1 ] as controls. For instance, dynamic controls at time t can be defined 
as the group {i : Tj > t } U {i : T{ < i, <5, = 2}. This leads to the estimation of only one 
ROC curve, for the event of interest. In |39j, Zheng et al. based their approach on 
initial estimates of the conditional cumulative incidence function for the event of in- 
terest. Their initial method provides consistent estimators if the proportional hazard 
assumption holds for each cause specific hazard. To relax this assumption a smooth 
estimator was also proposed. Another approach was described in 11211 . which fol- 
lows the lines of DeLong et al. |9). However, the suitability of this method to deal 
with censored data is not established. 

We shall add that, as pointed out in l28l[39l , employing a direct regression model 
for the conditional cumulative incidence would lead to a simpler estimation of the 
cumulative/dynamic AUC and a less convoluted interpretation of the marker effect. 
However, the extension to the setting of a longitudinal marker as well as the eval- 
uation of a risk score (which is usually built with a cause-specific hazard approach) 
would not be straightforward. 



6 Discussion 

While the AUC is uniquely defined in the context of the evaluation of diagnostic 
tests, its extension to prognostic markers has led to the development of a variety of 
definitions: these definitions vary according to the underlying definitions of cases 
(incident or cumulative) and controls (static or dynamic), and also depend on the 
study characteristics (the marker can be measured only once or repeatedly and com- 
peting events may be considered, or not). Regarding the choice of the retained defi- 
nition for cases and controls, no clear guidance has really emerged in the literature. 
It seems however that the cumulative/dynamic definition may be more appropriate 
for clinical decisions making (enrollment in clinical trials for instance) while the in- 
cident/static definition may be more appropriate for "pure" evaluation of the marker 
(if interpretation of trends of AUC values over time is of particular interest for in- 
stance). Once this definition has been chosen, appropriate estimators are available, 
depending on various assumptions (independence of the marker and censoring, pro- 
portional hazards, ...), and we presented most of them in this review article. 
For the sake of brevity, we were not able to cover some interesting extensions of 
time-dependent AUCs. In particular, covariate specific time-dependent ROC curves 
and AUCs have been studied in order to adjust the discrimination of a marker for 
external covariates (age, gender, ...). We refer the reader to |fl9l[33l for the standard 
setting, (H for the longitudinal setting and |39l for the competing events setting. In 
addition, some authors advocate that not the entire ROC curve is of interest and the 
area under only a portion of it should be computed, leading to the so-called partial 
AUC 111). In the context of prognostic markers, Hung and Chiang [20j| proposed a 
nonparametric estimator of the cumulative/dynamic time-dependent version of the 
partial AUC. Other interesting extensions include diverse censoring patterns l22l 
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(only right-censoring was considered in this review) and the combination of results 
from multiple studies [3| which is particularly useful in genomic studies. 
Another closely related topic is the evaluation of the added predictive ability of a 
new marker: for instance, we may wonder how better a risk score would be if we 
added some biological markers (SNPs, genes, ...). We refer the reader to the works 
in Il5l [l0ll23ll25ll for some insights, noticing though that most of these works do not 
cover the right-censored setting considered in our review. 



References 

1. M. Akritas. Nearest neighbor estimation of a bivariate distribution under random censoring. 
The Annals of Statistics, 22:1299-1327, 1994. 

2. P. Blanche, J. F. Dartigues, and H. Jacqmin-Gadda. Review and comparison of ROC curve 
estimators for a time-dependent outcome with marker-dependent censoring. Submitted, 2012. 

3. T. Cai, T.A. Gerds, Y. Zheng, and J. Chen. Robust prediction of t-year survival with data from 
multiple studies. Biometrics, 67:436^144, Jun 2011. 

4. T. Cai, M.S. Pepe, Y. Zheng, T. Lumley, and N.S. Jenny. The sensitivity and specificity of 
markers for event times. Biostatistics, 7: 182-197, Apr 2006. 

5. L.E. Chambless, CP. Cummiskey, and G. Cui. Several methods to assess improvement in risk 
prediction models: extension to survival analysis. Statistics in medicine, 30(l):22-38, 2011. 

6. L.E. Chambless and G. Diao. Estimation of time-dependent area under the ROC curve for 
long-term risk prediction. Statistics in Medicine, 25:3474-3486, Oct 2006. 

7. C.T Chiang and H. Hung. Nonparametric estimation for time-dependent AUC. Journal of 
Statistical Planning and Inference, 140(5): 1 162 - 1 174, 2010. 

8. G. Cortese and P. K. Andersen. Competing risks and time-dependent covariates. Biometrical 
Journal, 52(1): 138-158, 2010. 

9. E. R. DeLong, D. M. DeLong, and D. L. Clarke-Pearson. Comparing the areas under two or 
more correlated receiver operating characteristic curves: a nonparametric approach. Biomet- 
rics, 44(3):837-845, 1988. 

10. O.V. Demler, M.J. Pencina, and R.B. D'Agostino Sr. Equivalence of improvement in area 
under ROC curve and linear discriminant analysis coefficient under assumption of normality. 
Statistics in Medicine, 30(12): 1410-1418, 2011. 

11. L.E. Dodd and M.S. Pepe. Partial AUC estimation and regression. Biometrics, 59(3):614— 623, 
2003. 

12. R. Etzioni, M. Pepe, G. Longton, C. Hu, and G. Goodman. Incorporating the time dimension 
in receiver operating characteristic curves: a case study of prostate cancer. Medical Decision 
Making, 19(3):242-251, 1999. 

13. Y. Foucher, M. Giral, J.R Soulillou, and J.R Daures. Time-dependent ROC analysis for 
a three-class prognostic with application to kidney transplantation. Statistics in Medicine, 
29:3079-3087, Dec 2010. 

14. M.H. Gail and R.M. Pfeiffer. On criteria for evaluating models of absolute risk. Biostatistics, 
6(2):227-239, 2005. 

15. W. Gu and M.S. Pepe. Measures to summarize and compare the predictive capacity of markers. 
International Journal of Biostatistics, 5:27^19, 2009. 

16. P. J. Heagerty and M. S. Pepe. Semiparametric estimation of regression quantiles with ap- 
plication to standardizing weight for height and age in US children. Journal of the Royal 
Statistical Society: Series C (Applied Statistics), 48(4):533-551, 1999. 

17. P.J. Heagerty, T. Lumley, and M.S. Pepe. Time-dependent ROC curves for censored survival 
data and a diagnostic marker. Biometrics, 56:337-344, Apr 2000. 

18. P.J. Heagerty and Y. Zheng. Survival model predictive accuracy and ROC curves. Biometrics, 
61:92-105, 2005. 



12 



Paul Blanche, Aurelien Latouche, Vivian Viallon 



19. H. Hung and C.T. Chiang. Estimation methods for time-dependent AUC models with survival 
data. Canadian Journal of Statistics, 38(1):8— 26, 2010. 

20. H. Hung and C.T. Chiang. Nonparametric methodology for the time-dependent partial area 
under the ROC curve. Journal of Statistical Planning and Inference, 141(12):3829 - 3838, 
2011. 

21. M. Lee, K. A. Cronin, M. H. Gail, and E. J. Feuer. Predicting the absolute risk of dying from 
colorectal cancer and from other causes using population-based cancer registry data. Statistics 
in Medicine, 31(5):489-500, 2012. 

22. J. Li and S. Ma. Time-dependent ROC analysis under diverse censoring patterns. Statistics in 
Medicine, 30:1266-1277, 2011. 

23. M.J. Pencina, R.B. D'Agostino Sr, R.B. D'Agostino Jr, and R.S. Vasan. Evaluating the added 
predictive ability of a new marker: from area under the ROC curve to reclassification and 
beyond. Statistics in medicine, 27(2): 157-172, 2008. 

24. M.S. Pepe. The statistical evaluation of medical tests for classification and prediction. Oxford 
University Press, USA, 2004. 

25. M.S. Pepe, K.F. Kerr, G. Longton, and Z. Wang. Testing for improvement in prediction model 
performance. Preprint available at http://www.bepress.com/uwbiostat/paper379/, 201 1 . 

26. M.S. Pepe, Y. Zheng, Y. Jin, Y. Huang, C.R. Parikh, and W.C. Levy. Evaluating the ROC 
performance of markers for future events. Lifetime Data Analysis, 14:86-1 13, Mar 2008. 

27. D. Rizopoulos. Dynamic predictions and prospective accuracy in joint models for longitudinal 
and time-to-event data. Biometrics, 67:819-829, Sep 2011. 

28. P. Saha and P.J. Heagerty. Time-dependent predictive accuracy in the presence of competing 
risks. Biometrics, 66:999-1011, Dec 2010. 

29. P. Saha-Chaudhuri and PJ Heagerty. Non-parametric estimation of a time-dependent predic- 
tive accuracy curve. Biostatistics, 2012. 

30. GA. Satten and S. Datta. The kaplan-meier estimator as an inverse-probability-of-censoring 
weighted average. The American Statistician, 55(3):207-210, 2001. 

31. E.H. Slate and B.W. Turnbull. Statistical models for longitudinal biomarkers of disease onset. 
Statistics in medicine, 19(4):617-637, 2000. 

32. X. Song, X. H. Zhou, and S. Ma. Nonparametric receiver operating characteristic-based eval- 
uation for survival outcomes. Statistics in Medicine, 31(23):2660-2675, 2012. 

33. X. Song and X.H. Zhou. A semi-parametric approach for the covariate specific roc curve with 
survival outcome. Statistica Sinica, 18:947-965, 2008. 

34. F. Subtil, C. Pouteil-Noble, S. Toussaint, E. Villar, and M. Rabilloud. A Simple Modeling-free 
Method Provides Accurate Estimates of Sensitivity and Specificity of Longitudinal Disease 
Biomarkers. Methods of Information in Medicine, 48:299-305, 2009. 

35. A.A. Tsiatis and M. Davidian. Joint modeling of longitudinal and time-to-event data: an 
overview. Statistica Sinica, 14(3):809-834, 2004. 

36. H. Uno, T. Cai, L. Tian, and L.J Wei. Evaluating prediction rules for t-year survivors with 
censored regression models. Journal of the American Statistical Association, 102(478):527- 
537, 2007. 

37. V. Viallon and A. Latouche. Discrimination measures for survival outcomes: connection be- 
tween the AUC and the predictiveness curve. Biometrical Journal, 53:217-236, Mar 201 1. 

38. R. Xu and J. O'Quigley. Proportional hazards estimate of the conditional survival function. 
Journal of the Royal Statistical Society: Series B (Statistical Methodology), 62(4):667-680, 
2000. 

39. Y. Zheng, T. Cai, Y. Jin, and Z. Feng. Evaluating Prognostic Accuracy of Biomarkers under 
Competing Risk. Biometrics, Dec 2011. 

40. Y. Zheng and P.J. Heagerty. Semiparametric estimation of time-dependent ROC curves for 
longitudinal marker data. Biostatistics, 5(4):615-632, 2004. 

41. Y. Zheng and P.J. Heagerty. Partly conditional survival models for longitudinal data. Biomet- 
rics, 61(2):379-391, 2005. 

42. Y. Zheng and P.J. Heagerty. Prospective accuracy for longitudinal markers. Biometrics, 
63:332-341, 2007. 



